Generating event streams from encrypted network traffic monitored by remote capture agents

ABSTRACT

The disclosed embodiments provide a system that processes network data. During operation, the system obtains, at a remote capture agent, a first protocol classification for a first packet flow captured by the remote capture agent. Next, the system uses configuration information associated with the first protocol classification to build a first event stream from the first packet flow at the remote capture agent, wherein the first event stream comprises time-series event data generated from network packets in the first packet flow based on the first protocol classification. The system then transmits the first event stream over a network for subsequent storage and processing of the first event stream by one or more components on the network.

RELATED APPLICATION

This application claims benefit under 35 U.S.C. § 120 as a continuationof U.S. application Ser. No. 16/459,573, filed Jul. 1, 2019, which is acontinuation of U.S. application Ser. No. 15/665,268, filed Jul. 31,2017, issued as U.S. Pat. No. 10,382,599, which is a continuation ofU.S. application Ser. No. 14/528,898, filed Oct. 30, 2014, issued asU.S. Pat. No. 9,838,512, the entire contents of which are herebyincorporated by reference as if fully set forth herein. The applicant(s)hereby rescind any disclaimer of claim scope in the parentapplication(s) or the prosecution history thereof and advise the USPTOthat the claims in this application may be broader than any claim in theparent application(s).

The subject matter of this application is related to the subject matterin a non-provisional application by inventor Michael Dickey, entitled“Distributed Processing of Network Data Using Remote Capture Agents,”having Ser. No. 14/253,713, and filing date 15 Apr. 2014, issued as U.S.Pat. No. 10,127,273 (Attorney Docket No. 1015SP0055.01US).

The subject matter of this application is also related to the subjectmatter in a non-provisional application by inventors Vladimir A.Shcherbakov, Michael R. Dickey, Cary Glen Noel, Kishore R. Ramasayam andMignon L. Belongie, entitled “Streamlining Configuration ofProtocol-Based Network Data Capture by Remote Capture Agents,” havingSer. No. 14/528,932, and filed on 30 Oct. 2014 (Attorney Docket No. 1015SP0086.01US).

The subject matter of this application is also related to the subjectmatter in a non-provisional application by inventors Vijay Chauhan,Devendra M. Badhani, Luke K. Murphey and David Hazekamp, entitled“Capture Triggers for Capturing Network Data,” having Ser. No.14/528,918, and filed on 30 Oct. 2014, issued as U.S. Pat. No. 9,596,253(Attorney Docket No. 1015SP0087.01US).

BACKGROUND Field

The disclosed embodiments relate to techniques for processing networkdata. More specifically, the disclosed embodiments relate to techniquesfor performing protocol-based capture of network data using remotecapture agents in a distributed network environment.

Related Art

Over the past decade, the age of virtualization has triggered a seachange in the world of network data capture. Almost every networkcapture product available today is a physical hardware appliance thatcustomers have to purchase and configure. In addition, most network datacapture technologies are built from scratch to serve a specific purposeand address the needs of a particular vertical market. For example,network capture systems may be customized to extract data for securityand intrusion-detection purposes, collect network performance data,perform Quality of Service (QoS), redirect data, block network traffic,and/or perform other analysis or management of network traffic. Suchtargeted and/or fixed implementation and use of network capturetechnologies may preclude modification of the network capturetechnologies to address different and changing business needs.

Moreover, customers using conventional hardware-based network capturedevices typically connect the devices to other hardware devices in anetwork. The connections may allow the network capture devices to accessthe network and monitor network traffic between two or more points inthe network. Examples of such devices include a network Test AccessPoint (TAP) or Switched Port Analyzer (SPAN) port. After the networktraffic is captured, cumbersome Extraction, Transform, and Load (“ETL”)processes may be performed to filter, transform, and/or aggregate datafrom the network traffic and enable the extraction of business valuefrom the data.

However, customers are moving away from managing physical servers anddata centers and toward public and private cloud computing environmentsthat provide software, hardware, infrastructure, and/or platformresources as hosted services using computing, storage, and/or networkdevices at remote locations. For these customers, it is eitherimpossible, or at best extremely challenging, to deploy physical networkcapture devices and infrastructure in the cloud computing environments.

Consequently, network data capture may be facilitated by mechanisms forstreamlining the deployment and configuration of network capturetechnology at distributed and/or remote locations.

SUMMARY

The disclosed embodiments provide a system that processes network data.During operation, the system obtains, at a remote capture agent, a firstprotocol classification for a first packet flow captured by the remotecapture agent. Next, the system uses configuration informationassociated with the first protocol classification to build a first eventstream from the first packet flow at the remote capture agent, whereinthe first event stream includes time-series event data generated fromnetwork packets in the first packet flow based on the first protocolclassification. The system then transmits the first event stream over anetwork for subsequent storage and processing of the first event streamby one or more components on the network.

In some embodiments, the system also obtains, at the remote captureagent, the configuration information from a configuration server overthe network. Next, the system uses the configuration information toconfigure the generation of the time-series event data from the networkpackets during runtime of the remote capture agent.

In some embodiments, the system also obtains, at the remote captureagent, a second protocol classification for a second packet flowcaptured at the remote capture agent. Next, the system usesconfiguration information associated with the second protocolclassification to build a second event stream from the second packetflow at the remote capture agent, wherein the second event streamincludes time-series event data from network packets in the secondpacket flow based on the second protocol classification. The system thentransmits the second event stream over the network.

In some embodiments, the system also identifies, at the remote captureagent, the network packets in the first packet flow based on controlinformation in the network packets.

In some embodiments, the system also assembles the first packet flowfrom the network packets. Upon detecting encryption of the networkpackets in the first packet flow, the system decrypts the networkpackets in the first packet flow prior to obtaining the first protocolclassification for the first packet flow.

In some embodiments, the network packets in the first packet flow areassociated with at least one of a source, a destination, a networkaddress, a port, and a transport layer protocol.

In some embodiments, using the configuration information associated withthe first protocol classification to build the first event stream fromthe first packet flow at the remote capture agent includes:

-   -   obtaining one or more event attributes associated with the first        protocol classification from the configuration information;    -   (ii) extracting the one or more event attributes from the        network packets in the first packet flow;    -   (iii) using the configuration information to transform the        extracted one or more event attributes; and    -   (iv) including the transformed one or more event attributes in        the first event stream.

In some embodiments, the first protocol classification comprises atleast one of a transport layer protocol, a session layer protocol, apresentation layer protocol, and an application layer protocol.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows a schematic of a system in accordance with the disclosedembodiments.

FIG. 2A shows a remote capture agent in accordance with the disclosedembodiments.

FIG. 2B shows the protocol-based capture of network data using a remotecapture agent in accordance with the disclosed embodiments.

FIG. 3 shows a configuration server in accordance with the disclosedembodiments.

FIG. 4A shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 4B shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 4C shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 4D shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 4E shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 4F shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 5A shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 5B shows an exemplary screenshot in accordance with the disclosedembodiments.

FIG. 6 shows a flowchart illustrating the processing of network data inaccordance with the disclosed embodiments.

FIG. 7 shows a flowchart illustrating the process of using configurationinformation associated with a protocol classification to build an eventstream from a packet flow in accordance with the disclosed embodiments.

FIG. 8 shows a flowchart illustrating the process of facilitating theprocessing of network data in accordance with the disclosed embodiments.

FIG. 9 shows a flowchart illustrating the process of facilitating theprocessing of network data in accordance with the disclosed embodiments.

FIG. 10 shows a computer system in accordance with the disclosedembodiments.

In the figures, like reference numerals refer to the same figureelements.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the embodiments, and is provided in the contextof a particular application and its requirements. Various modificationsto the disclosed embodiments will be readily apparent to those skilledin the art, and the general principles defined herein may be applied toother embodiments and applications without departing from the spirit andscope of the present disclosure. Thus, the present invention is notlimited to the embodiments shown, but is to be accorded the widest scopeconsistent with the principles and features disclosed herein.

The data structures and code described in this detailed description aretypically stored on a computer-readable storage medium, which may be anydevice or medium that can store code and/or data for use by a computersystem. The computer-readable storage medium includes, but is notlimited to, volatile memory, non-volatile memory, magnetic and opticalstorage devices such as disk drives, magnetic tape, CDs (compact discs),DVDs (digital versatile discs or digital video discs), or other mediacapable of storing code and/or data now known or later developed.

The methods and processes described in the detailed description sectioncan be embodied as code and/or data, which can be stored in acomputer-readable storage medium as described above. When a computersystem reads and executes the code and/or data stored on thecomputer-readable storage medium, the computer system performs themethods and processes embodied as data structures and code and storedwithin the computer-readable storage medium.

Furthermore, methods and processes described herein can be included inhardware modules or apparatus. These modules or apparatus may include,but are not limited to, an application-specific integrated circuit(ASIC) chip, a field-programmable gate array (FPGA), a dedicated orshared processor that executes a particular software module or a pieceof code at a particular time, and/or other programmable-logic devicesnow known or later developed. When the hardware modules or apparatus areactivated, they perform the methods and processes included within them.

The disclosed embodiments provide a method and system for facilitatingthe processing of network data. As shown in FIG. 1, the network data maybe captured using a data-processing system 100 in a distributed networkenvironment. In the illustrated embodiment, system 100 includes a set ofconfiguration servers 120 in communication with a set of remote captureagents 151-153 over one or more networks 190.

Although system 100 only depicts three configuration servers 120 andthree remote capture agents 151-153, any number of configuration servers120 and/or remote capture agents 151-153 may be configured to operateand/or communicate with one another within the data-processing system.For example, a single physical and/or virtual server may perform thefunctions of configuration servers 120. Alternatively, multiple physicaland/or virtual servers or network elements may be logically connected toprovide the functionality of configuration servers 120. Theconfiguration server(s) may direct the activity of multiple distributedremote capture agents 151-153 installed on various client computingdevices across one or more networks. In turn, remote capture agents151-153 may be used to capture network data from multiple remote networkdata sources.

Further, embodiments described herein can be configured to capturenetwork data in a cloud-based environment, such as cloud 140 depicted inthe illustrated embodiment, and to generate events such as timestampedrecords of network activity from the network data. Remote capture agents151-153 may capture network data originating from numerous distributednetwork servers, whether they are physical hardware servers or virtualmachines running in cloud 140. In cloud-based implementations, remotecapture agents 151-153 will generally only have access to informationthat is communicated to and received from machines running in thecloud-based environment. This is because, in a cloud environment, thereis generally no access to any of the physical network infrastructure, ascloud computing may utilize a “hosted services” delivery model where thephysical network infrastructure is typically managed by a third party.

Embodiments further include the capability to separate the data capturetechnology into a standalone component that can be installed directly onclient servers, which may be physical servers or virtual machinesresiding on a cloud-based network (e.g., cloud 140), and used to captureand generate events for all network traffic that is transmitted in andout of the client servers. This eliminates the need to deploy andconnect physical hardware to network TAPS or SPAN ports, thus allowingusers to configure and change their data capture configurationon-the-fly rather than in fixed formats.

In the illustrated embodiment, remote capture agents 152-153 are incommunication with network servers 130 residing in cloud 140, and remotecapture agent 151 is located in cloud 140. Cloud 140 may represent anynumber of public and private clouds, and is not limited to anyparticular cloud configuration. Network servers 130 residing in cloud140 may be physical servers and/or virtual machines in cloud 140, andnetwork traffic to and from network servers 130 may be monitored byremote capture agent 151 and/or other remote capture agents connected tonetwork servers 130. Further, remote capture agents 152-153 may also runin cloud 140 on physical servers and/or virtual machines. Those skilledin the art will appreciate that any number of remote capture agents maybe included inside or outside of cloud 140.

Remote capture agents 151-153 may analyze network packets received fromthe networks(s) to which remote capture agents 151-153 are connected toobtain network data from the network packets and generate a number ofevents from the network data. For example, each remote capture agent151-153 may listen for network traffic on network interfaces availableto the remote capture agent. Network packets transmitted to and/or fromthe network interfaces may be intercepted by the remote capture agentand analyzed, and relevant network data from the network packets may beused by the remote capture agent to create events related to the networkdata. Such events may be generated by aggregating network data frommultiple network packets, or each event may be generated using thecontents of only one network packet. A sequence of events from a remotecapture agent may then be included in one or more event streams that areprovided to other components of system 100.

Configuration servers 120, data storage servers 135, and/or othernetwork components may receive event data (e.g., event streams) fromremote capture agents 151-153 and further process the event data beforethe event data is stored by data storage servers 135. In the illustratedembodiment, configuration servers 120 may transmit event data to datastorage servers 135 over a network 101 such as a local area network(LAN), wide area network (WAN), personal area network (PAN), virtualprivate network, intranet, mobile phone network (e.g., a cellularnetwork), WiFi network, Ethernet network, and/or other type of networkthat enables communication among computing devices. The event data maybe received over a network (e.g., network 101, network 190) at one ormore event indexers (see FIG. 10) associated with data storage servers135.

In addition, system 100 may include functionality to determine the typesof network data collected and/or processed by each remote capture agent151-153 to avoid data duplication at the indexers, data storage servers135, and/or other components of system 100. For example, remote captureagents 152-153 may process network traffic from the same network.However, remote capture agent 152 may generate page view events from thenetwork traffic, and remote capture agent 153 may generate requestevents (e.g., of HyperText Transfer Protocol (HTTP) requests andresponses) from the network traffic.

In one or more embodiments, configuration servers 120 includeconfiguration information that is used to configure the creation ofevents from network data on remote capture agents 151-153. In addition,such configuration may occur dynamically during event processing (e.g.,at runtime). Conversely, because most conventional network capturetechnologies target specific end uses, they have been designed tooperate in a fixed way and generally cannot be modified dynamically oreasily to address different and changing business needs.

At least certain embodiments described herein are adapted to provide adistributed remote capture platform in which the times at which eventsare communicated to the configuration servers 120 and the fields to beincluded in the events are controlled by way of user-modifiableconfiguration rather than by “hard coding” fixed events withpre-determined fields for a given network capture mechanism. The remoteconfiguration capability described herein also enables additionalin-memory processing (e.g., filtering, transformation, normalization,aggregation, etc.) on events at the point of capture (e.g., remotecapture agents 151-153) before the events are transmitted to othercomponents of system 100.

Configuration information stored at each configuration server 120 may becreated and/or updated manually at the configuration server and/or at anetwork element in communication with the configuration server. Forexample, a user may upload a configuration file containing configurationinformation for a remote capture agent to one or more configurationservers 120 for subsequent propagation to the remote capture agent.Alternatively, the user may use a GUI to provide the configurationinformation, as described in further detail below with respect to FIGS.4A-4E. The configuration information may further be provided by one ormore applications running on a separate server or network element, suchas data storage servers 135.

Remote capture agents 151-153 may then use the configuration informationto generate events from captured network packets. When changes in theconfiguration information at the configuration server are detected atthe remote capture agents, logic in the remote capture agents may beautomatically reconfigured in response. This means the remote captureagents may be configured dynamically to produce different events,transform the events, and/or communicate event streams to differentcomponents of system 100.

To detect changes in configuration information at configuration servers120, remote capture agents 151-153 may poll configuration servers 120 atperiodic intervals for updates to the configuration information. Theupdates may then be pulled from configuration servers 120 by remotecapture agents 151-153. Conversely, updates to the configurationinformation may be pushed from configuration servers 120 to remotecapture agents 151-153 at periodic intervals and/or when changes to theconfiguration information have been made.

In one embodiment, configuration servers 120 include a list of eventstreams generated by remote capture agents 151-153, as well as theconfiguration information used to generate the event streams at remotecapture agents 151-153. The configuration information may include aunique identifier for each event stream, the types of events to beincluded in the event stream, one or more fields to be included in eachevent, and/or one or more filtering rules for filtering events to beincluded in the event stream. Using configuration information todynamically modify network data capture by remote capture agents (e.g.,remote capture agents 151-153) is described in a non-provisionalapplication by inventor Michael Dickey, entitled “Distributed Processingof Network Data Using Remote Capture Agents,” having Ser. No.14/253,713, and filing date 15 Apr. 2014, issued as U.S. Pat. No.10,127,273, which is incorporated herein by reference.

In one or more embodiments, system 100 includes functionality to performprotocol-based capture and analysis of network data using remote captureagents 151-153. First, remote capture agents 151-153 may be configuredto generate event streams from packet flows captured at remote captureagents 151-153 based on protocol classifications for the packet flows,as described below with respect to FIGS. 2A-2B. Second, configurationservers 120 may include functionality to streamline the configuration ofremote capture agents 151-153 in generating protocol-specific eventstreams, as described below with respect to FIGS. 3A-3B and 4A-4C.Third, configuration servers 120 and/or remote capture agents 151-153may enable the use of capture triggers to capture additional networkdata based on the identification of potential security risks frompreviously generated event streams, as described below with respect toFIGS. 3A, 3C and 5A-5C.

FIG. 2A shows a remote capture agent 250 in accordance with thedisclosed embodiments. In the illustrated embodiment, remote captureagent 250 is adapted to receive configuration information from one ormore configuration servers 120 over network 101. Remote capture agent250 may be installed at a customer's premises on one or more of thecustomer's computing resources. Remote capture agent 250 may also beinstalled in a remote computing environment such as a cloud computingsystem. For example, remote capture agent 250 may be installed on aphysical server and/or in a virtual computing environment (e.g., virtualmachine) that is distributed across one or more physical machines.

Remote capture agent 250 includes a communications component 203configured to communicate with network elements on one or more networks(e.g., network 101) and send and receive network data (e.g., networkpackets) over the network(s). As depicted, communications component 203may communicate with configuration servers 120 over network 101.Communications component 203 may also communicate with one or moresources of network data, such as network servers 130 of FIG. 1.

Network data received at communications component 203 may be captured bya capture component 205 coupled with communications component 203.Capture component 205 may capture some or all network data fromcommunications component 203. For example, capture component 205 maycapture network data based on the sources and/or destinations of thenetwork data, the types of network data, the protocol associated withthe network data, and/or other characteristics of the network data.

In addition, the network data may be captured based on configurationinformation stored in a configuration component 204 of remote captureagent 250. As mentioned above, the configuration information may bereceived from configuration servers 120 over network 101. Theconfiguration information may then be used to dynamically configure orreconfigure remote capture agent 250 in real-time. For example, newlyreceived configuration information in configuration component 204 may beused to configure the operation of remote capture agent 250 duringprocessing of events from network data by remote capture agent 250.

To dynamically configure remote capture agent 250, configurationinformation received by configuration component 204 from configurationservers 120 may be provided to other components of remote capture agent250. More specifically, remote capture agent 250 includes an eventsgenerator 207 that receives network data from network data capturecomponent 205 and generates events from the network data based onconfiguration information from configuration component 204.

Using configuration information provided by configuration servers 120,remote capture agent 250 can be instructed to perform any number ofevent-based processing operations. For example, the configurationinformation may specify the generation of event streams associated withnetwork (e.g., HTTP, Simple Mail Transfer Protocol (SMTP), Domain NameSystem (DNS)) transactions, business transactions, errors, alerts,clickstream events, and/or other types of events. The configurationinformation may also describe custom fields to be included in theevents, such as values associated with specific clickstream terms. Theconfiguration information may include additional parameters related tothe generation of event data, such as an interval between consecutiveevents and/or the inclusion of transactions and/or errors matching agiven event in event data for the event. Configuration information forconfiguring the generation of event streams from network data capturedby remote capture agents is further described in the above-referencedapplication.

An events transformer 209 may further use the configuration informationto transform some or all of the network data from capture component 205and/or events from events generator 207 into one or more sets oftransformed events. In one or more embodiments, transformationsperformed by events transformer 209 include aggregating, filtering,cleaning, and/or otherwise processing events from events generator 207.Configuration information for the transformations may thus include anumber of parameters that specify the types of transformations to beperformed, the types of data on which the transformations are to beperformed, and/or the formatting of the transformed data.

A rules comparison engine 208 in remote capture agent 250 may receiveevents from event generator 207 and compare one or more fields from theevents to a set of filtering rules in the configuration information todetermine whether to include the events in an event stream. For example,the configuration information may specify packet-level, protocol-level,and/or application-level filtering of event data from event streamsgenerated by remote capture agent 250.

Finally, a data enrichment component 211 may further transform eventdata to a different form or format based on the configurationinformation from configuration component 204. For example, dataenrichment component 211 may use the configuration information tonormalize the data so that multiple representations of the same value(e.g., timestamps, measurements, etc.) are converted into the same valuein transformed event data.

Data can be transformed by data enrichment component 211 in any numberof ways. For example, remote capture agent 250 may reside on a clientserver in Cupertino, California, where all the laptops associated withthe client server have been registered with the hostname of the clientserver. Remote capture agent 250 may use the registration data to lookup an Internet Protocol (IP) address in a look-up table (LUT) that isassociated with one or more network elements of the client server'slocal network. Remote capture agent 250 may then resolve a user's IPaddress into the name of the user's laptop, thereby enabling inclusionof the user's laptop name in transformed event data associated with theIP address. The transformed event data may then be communicated toconfiguration servers 120 and/or a central transformation serverresiding in San Francisco for further processing, indexing, and/orstorage.

As mentioned above, remote capture agent 250 may perform protocol-basedgeneration of event streams from network data. As shown in FIG. 2B,configuration component 204 may obtain protocol-specific configurationinformation (e.g., protocol-specific configuration information A 212,protocol-specific configuration information B 214) from one or moreconfiguration servers (e.g., configuration servers 120). For example,configuration information from the configuration server(s) may betransmitted over network 101 to communications component 203, whichprovides the configuration information to configuration component 204for storage and/or further processing.

Protocol-specific configuration information from configuration component204 may be used to configure the generation of event streams (e.g.,event stream C 232, event stream D 234, event stream E 240, event streamF 242) based on protocol classifications of network packets (e.g.,network packets C 216, network packets D 218) captured by capturecomponent 205. For example, protocol-specific configuration informationfrom configuration component 204 may specify the creation of eventstreams from the network packets based on the protocols used in thenetwork packets, such as HTTP, DNS, SMTP, File Transfer Protocol (FTP),Server Message Block (SMB), Network File System (NFS), Internet ControlMessage Protocol (ICMP), email protocols, database protocols, and/orsecurity protocols. Such event streams may include event attributes thatare of interest to the respective protocols.

Before the event streams are generated from the network packets, capturecomponent 205 may assemble the network packets into one or more packetsflows (e.g., packet flow C 220, packet flow D 222). First, capturecomponent 205 may identify the network packets in a given packet flowbased on control information in the network packets. The packet flow mayrepresent a communication path between a source and a destination (e.g.,host, multicast group, broadcast domain, etc.) on the network. As aresult, capture component 205 may identify network packets in the packetflow by examining network (e.g., IP) addresses, ports, sources,destinations, and/or transport protocols (e.g., Transmission ControlProtocol (TCP), User Datagram Protocol (UDP), etc.) from the headers ofthe network packets.

Next, capture component 205 may assemble the packet flow from thenetwork packets. For example, capture component 205 may assemble a TCPpacket flow by rearranging out-of-order TCP packets. Conversely, capturecomponent 205 may omit reordering of the network packets in the packetflow if the network packets use UDP and/or another protocol that doesnot provide for ordered packet transmission.

After the packet flow is assembled, capture component 205 and/or anothercomponent of remote capture agent 250 may detect encryption of thenetwork packets in the packet flow by analyzing the byte signatures ofthe network packets' payloads. For example, the component may analyzethe network packets' payloads for byte signatures that are indicative ofSecure Sockets Layer (SSL) and/or Transport Layer Security (TLS)encryption. If the network packets are detected as encrypted, thecomponent may decrypt the network packets. For example, the componentmay have access to private keys from an SSL server used by the networkflow and perform decryption of the network packets to obtain plaintextpayload data in the order in which the data was sent. Such access toprivate keys may be given to remote capture agent 250 by anadministrator associated with the network flow, such as an administratorof the host from which the network packets are transmitted.

Events generator 207 may then obtain a protocol classification (e.g.,protocol classification C 224, protocol classification D 226) for eachpacket flow identified, assembled, and/or decrypted by capture component205. For example, events generator 207 may use a protocol-decodingmechanism to analyze the headers and/or payloads of the network packetsin the packet flow and return protocol identifiers of one or moreprotocols used in the network packets. The protocol-decoding mechanismmay additionally provide metadata related to the protocols, such asmetadata related to traffic volume, application usage, applicationperformance, user and/or host identifiers, content (e.g., media, files,etc.), and/or file metadata (e.g., video codecs and bit rates).

Once the protocol classification is obtained for a packet flow, eventsgenerator 207 may use protocol-specific configuration informationassociated with the protocol classification from configuration component204 to build an event stream (e.g., event stream C 232, event stream D234) from the packet flow. As mentioned above and in theabove-referenced application, the event stream may include time-seriesevent data generated from network packets in the packet flow. To createthe event stream, events generator 207 may obtain one or more eventattributes associated with the protocol classification from theconfiguration information. Next, event generator 207 may extract theevent attribute(s) from the network packets in the first packet flow.Events generator 207 may then include the extracted event attribute(s)in the event stream.

For example, events generator 207 may obtain a protocol classificationof DNS for a packet flow from capture component 205 andprotocol-specific configuration information for generating event streamsfrom DNS traffic from configuration component 204. The protocol-specificconfiguration information may specify the collection of event attributessuch as the number of bytes transferred between the source anddestination, network addresses and/or identifiers for the source anddestination, DNS message type, DNS query type, return message, responsetime to a DNS request, DNS transaction identifier, and/or a transportlayer protocol. In turn, events generator 207 may parse theprotocol-specific configuration to identify the event attributes to becaptured from the packet flow. Next, events generator 207 may extractthe specified event attributes from the network packets in the packetflow and/or metadata received with the protocol classification of thepacket flow and generate time-stamped event data from the extractedevent attributes. Events generator 207 may then provide the time-stampedevent data in an event stream to communications component 203 fortransmission of the event stream over a network to one or moreconfiguration servers, data storage servers, indexers, and/or othercomponents for subsequent storage and processing of the event stream bythe component(s).

As described above and in the above-referenced application, network datafrom capture component 205 and/or event data from events generator 207may be transformed by events transformer 209 into transformed event datathat is provided in lieu of or in addition to event data generated byevents generator 207. For example, events transformer 209 may aggregate,filter, clean, and/or otherwise process event attributes from eventsgenerator 207 to produce one or more sets of transformed eventattributes (e.g., transformed event attributes 1 236, transformed eventattributes z 238). Events transformer 209 may then include thetransformed event attributes into one or more additional event streams(e.g., event stream 1 240, event stream z 242) that may be transmittedover the network for subsequent storage and processing of the eventstream(s) by other components on the network. Such transformation ofevent data at remote capture agent 250 may offload subsequent processingof the event data at configuration servers and/or other components onthe network. Moreover, if the transformation reduces the size of theevent data (e.g., by aggregating the event data), network trafficbetween remote capture agent 250 and the other components may bereduced, along with the storage requirements associated with storing theevent data at the other components.

As with protocol-based generation of event data by events generator 207,events transformer 209 may use protocol-specific configurationinformation from configuration component 204 to transform network and/orevent data from a given packet flow and/or event stream. For example,events transformer 209 may obtain protocol-specific configurationinformation for aggregating HTTP events and use the configurationinformation to generate aggregated HTTP events from HTTP events producedby events generator 207. The configuration information may include oneor more key attributes used to generate a unique key representing anaggregated event from the configuration information. For example, keyattributes for generating an aggregated HTTP event may include thesource and destination IP addresses and ports in a set of HTTP events. Adifferent unique key and aggregated HTTP event may thus be generated foreach unique combination of source and destination IP addresses and portsin the HTTP events.

The configuration information may also specify one or more aggregationattributes to be aggregated prior to inclusion in the aggregated event.For example, aggregation attributes for generating an aggregated HTTPevent from HTTP event data may include the number of bytes and packetssent in each direction between the source and destination. Datarepresented by the aggregation attributes may be included in theaggregated HTTP event by summing, averaging, and/or calculating asummary statistic from the number of bytes and packets sent in eachdirection between the source and destination. Aggregation of event datais described in further detail below with respect to FIG. 4C.

FIG. 3 shows a configuration server 320 in accordance with the disclosedembodiments. As shown in the illustrated embodiment, configurationserver 320 is in communication with multiple remote capture agents 350over network 190, and remote capture agents 350 are distributedthroughout network 190 and cloud 140. Configuration server 320 includesa communications component 303 that receives events from remote captureagents 350 over networks 190 and/or 140. Communications component 303may also communicate with one or more data storage servers, such as datastorage servers 135 of FIG. 1.

Configuration server 320 also includes a configuration component 304that stores configuration information for remote capture agents 350. Asdescribed above, the configuration information may specify the types ofevents to produce, data to be included in the events, and/ortransformations to be applied to the data and/or events to producetransformed events. Some or all of the transformations may be specifiedin a set of filtering rules 321 that may be applied to event data atremote capture agents 350 to determine a subset of the event data to beincluded in one or more event streams that are sent to configurationserver 320 and/or other components.

Configuration server 320 may also include a data processing component311 that performs additional processing of the event streams based onconfiguration information from configuration component 304. As discussedin the above example with respect to FIG. 2, event data may betransformed at a remote capture agent (e.g., remote capture agent 250)during resolution of the user's IP address was into the name of theuser's laptop. The transformed event data may be sent to configurationserver 320 and/or a transformation server for additional processingand/or transformation, such as taking the host name from the transformedevent data, using an additional LUT to obtain a user identifier (userID) of the person to which the laptop is registered, and furthertransforming the event data by including the user ID in the event databefore forwarding the event data to a third server (e.g., atransformation server) for another round of processing.

Configuration server 320 may also provide a GUI 325 that can be used toconfigure or reconfigure the information contained in configurationcomponent 304. The operation of GUI 325 is discussed in further detailbelow with respect to FIGS. 4A-4E and 5A-5C.

Finally, configuration server 320 may provide a risk-identificationmechanism 307 for identifying a security risk from time-series eventdata generated by remote capture agents 350, as well as a capturetrigger 309 for generating additional time-series event data based onthe security risk. For example, risk-identification mechanism 307 mayallow a user to view and/or search for events that may representsecurity risks through GUI 325. Risk-identification mechanism 307 and/orGUI 325 may also allow the user to set and/or activate capture trigger309 based on the events shown and/or found through risk-identificationmechanism 307 and/or GUI 325.

In particular, risk-identification mechanism 307 and/or GUI 325 mayallow the user to manually activate capture trigger 309 afterdiscovering a potential security risk. In turn, the activated capturetrigger 309 may modify configuration information in configurationcomponent 304 that is propagated to remote capture agents 350 to triggerthe capture of additional network data by remote capture agents 350.

Alternatively, risk-identification mechanism 307 may allow the user tocreate a search and/or recurring search for time-series event data thatmay match a security risk. If the search and/or recurring search findstime-series event data that matches the security risk, capture trigger309 may automatically be activated to enable the generation ofadditional time-series event data, such as event data containing one ormore attributes associated with one or more protocols that facilitateanalysis of the security risk. Such automatic activation of capturetrigger 309 may allow the additional event data to be generatedimmediately after a notable event is detected, thus averting the loss ofcaptured network data that results from enabling additional network datacapture only after a potential security risk is manually identified(e.g., by an analyst). Triggering the generation of additionaltime-series event data from network packets on remote agents based onpotential security risks is described in further detail below withrespect to FIGS. 5A-5C.

FIG. 4A shows an exemplary screenshot in accordance with the disclosedembodiments. More specifically, FIG. 4A shows a screenshot of a GUI,such as GUI 325 of FIG. 3. As described above, the GUI may be used toobtain configuration information that is used to configure thegeneration of event streams containing time-series event data at one ormore remote capture agents distributed across a network.

As shown in FIG. 4A, the GUI includes a table with a set of columns402-408 containing high-level information related to event streams thatmay be created using the configuration information. Each row of thetable may represent an event stream, and rows of the table may be sortedby column 402.

Column 402 shows an alphabetized list of names of the event streams, andcolumn 404 provides descriptions of the event streams. For example,columns 402-404 may include names and descriptions of event streamsgenerated from HTTP, Dynamic Host Configuration Protocol (DHCP), DNS,FTP, email protocols, database protocols, NFS, Secure Message Block(SMB), security protocols, Session Initiation Protocol (SIP), TCP,and/or UDP network traffic. Columns 402-404 may thus indicate that eventstreams may be generated based on transport layer protocols, sessionlayer protocols, presentation layer protocols, and/or application layerprotocols.

A user may select a name of an event stream under column 402 to accessand/or update configuration information for configuring the generationof the event stream. For example, the user may select “DemoHTTP” incolumn 402 to navigate to a screen of the GUI that allows the user tospecify event attributes, filters, and/or aggregation informationrelated to creating the “DemoHTTP” event stream, as discussed in furtherdetail below with respect to FIGS. 4B-4E.

Column 406 specifies whether each event stream is enabled or disabled.For example, column 406 may indicate that the “AggregateHTTP,”“DemoHTTP,” “dns,” “ftp,” “mysql-query,” “sip,” “tcp,” and “udp” eventstreams are enabled. If an event stream is enabled, time-series eventdata may be included in the event stream based on the configurationinformation for the event stream.

Column 408 specifies whether each event stream is cloned from anexisting event stream. For example, column 408 may indicate that the“AggregateHTTP” and “DemoHTTP” event streams have been cloned (e.g.,copied) from other event streams, while the remaining event streams maybe predefined with default event attributes.

The GUI also includes a user-interface element 410 (e.g., “CloneStream”). A user may select user-interface element 410 to create a newevent stream as a copy of an event stream listed in the GUI. Afteruser-interface element 410 is selected, an overlay may be displayed thatallows the user to specify a name for the new event stream, adescription of the new event stream, and an existing event stream fromwhich the new event stream is to be cloned. The new event stream maythen be created with the same event attributes and/or configurationoptions as the existing event stream, and the user may use the GUI tocustomize the new event stream as a variant of the existing event stream(e.g., by adding or removing event attributes, filters, and/oraggregation information).

FIG. 4B shows an exemplary screenshot in accordance with the disclosedembodiments. More specifically, FIG. 4B shows a screenshot of the GUI ofFIG. 4A after the user has selected “DemoHTTP” from column 402. Inresponse to the selection, the GUI displays configuration informationand/or configuration options for the “DemoHTTP” event stream.

Like the GUI of FIG. 4A, the GUI of FIG. 4B may include a table. Eachrow in the table may represent an event attribute that is eligible forinclusion in the event stream. For example, an event attribute may beincluded in the table if the event attribute can be obtained fromnetwork packets that include the protocol of the event stream. Columns412-420 of the table may allow the user to use the event attributes togenerate time-series event data that is included the event stream.First, column 412 includes a series of checkboxes that allows the userto include individual event attributes in the event stream or excludethe event attributes from the event stream. If a checkbox is checked,the corresponding event attribute is added to the event stream, and therow representing the event attribute is shown with other included eventattributes in an alphabetized list at the top of the table. If acheckbox is not checked, the corresponding event attribute is omittedfrom the event stream, and the row representing the event attribute isshown with other excluded event attributes in an alphabetized listfollowing the list of included event attributes. Those skilled in theart will appreciate that the GUI may utilize other sortings and/orrankings of event attributes in columns 412-420.

Columns 414-418 may provide information related to the event attributes.Column 414 may show the names of the event attributes, column 416 mayprovide a description of each event attribute, and column 418 mayprovide a term representing the event attribute. In other words, columns414-418 may allow the user to identify the event attributes and decidewhether the event attributes should be included in the event stream.

Column 420 may include a series of links labeled “Add.” The user mayselect one of the links to access a portion of the GUI that allows theuser to set a filter for the corresponding event attribute. The filtermay then be used in the generation of the event stream from networkdata. Creation of filters for generating event streams from networkpackets is described in further detail below with respect to FIGS.4D-4E.

The GUI of FIG. 4B also includes information 422 related to the eventstream. For example, information 422 may include the name (e.g.,“DemoHTTP”) of the event stream, the protocol classification and/or type(e.g., “http.event”) of the event stream, and the number of filters(e.g., “0 filters configured”) set for the event stream. Information 422may also include a checkbox 436 that identifies if the event streamcontains aggregated event data. If checkbox 436 is checked, the GUI maybe updated with options associated with configuring the generation of anaggregated event stream, as described below with respect to FIG. 4C.

Finally, the GUI of FIG. 4B includes a set of user-interface elements424-434 for managing the event stream. First, the user may selectuser-interface element 424 (e.g., “Enabled”) to enable generation of theevent stream from network data and user-interface element 426 (e.g.,“Disabled”) to disable the generation of the event stream from thenetwork data.

Next, the user may select user-interface element 428 (e.g., “Clone”) toclone the event stream and user-interface element 430 (e.g., “Delete”)to delete the event stream. If the user selects user-interface element428, the GUI may obtain a name and description for the cloned eventstream from the user. Next, the GUI may copy the content of columns412-420, including configuration options (e.g., checkboxes in column 412and filters added using links in column 420) that have been changed butnot yet saved by the user, to a new screen for configuring thegeneration of the cloned event stream.

If the user selects user-interface element 430, the GUI may remove theevent stream from the table in FIG. 4A. In turn, a representation of theevent stream may be removed from the configuration information to stopthe generation of time-series event data in the event stream by one ormore remote capture agents.

The user may select user-interface element 432 (e.g., “Cancel”) todischarge changes to the configuration information made in the currentscreen of the GUI. Conversely, the user may select user-interface 434(e.g., “Save”) to propagate the changes to the configurationinformation, and in turn, update the generation of event data fromnetwork packets captured by the remote capture agents based on thechanges.

FIG. 4C shows an exemplary screenshot in accordance with the disclosedembodiments. In particular, FIG. 4C shows a screenshot of the GUI ofFIG. 4B after checkbox 436 has been checked. Because checkbox 436 ischecked, the GUI includes a number of user-interface elements forconfiguring the generation of an aggregated event stream. The aggregatedevent stream may include aggregated event data, which in turn may begenerated by aggregating and/or extracting event attributes from one ormore network packets in a packet flow. For example, an HTTP event may begenerated from one to several HTTP packets representing an HTTPrequest/response pair. Event attributes from multiple HTTP events maythen be aggregated into a single aggregated HTTP event to reduce theamount of event data generated from the network data without losingimportant attributes of the event data.

As shown in FIG. 4C, a new column 438 is added to the table. Each row incolumn 438 may include a pair of user-interface elements (e.g., buttons)that allow the user to identify the corresponding event attribute as akey attribute or an aggregation attribute. One or more key attributesmay be used to generate a unique key representing each aggregated event,and one or more aggregation attributes may be aggregated prior toinclusion in the aggregated event. Some event attributes (e.g.,“dest_ip,” “src_ip,” “uri_path”) may only be used as key attributesbecause the event attributes are not numeric in nature. On the otherhand, event attributes that may be summed (e.g., “dest_port,” “status,”“bytes,” “bytes_in,” “bytes_out,” “time_taken”) may have numeric values.

Event attributes identified as key attributes in column 438 may besorted at the top of the table, followed by event attributes identifiedas aggregation attributes. Event attributes that are not included in theevent stream (e.g., event attributes with unchecked checkboxes in column412) may be shown below the aggregation attributes in the table.Alternatively, event attributes may be displayed in the table accordingto other sortings and/or rankings.

While sums are the only type of aggregation shown in the GUI of FIG. 4C,other types of aggregation may also be used to generate aggregated eventdata. For example, aggregated event streams may be created usingminimums, maximums, averages, standard deviations, and/or other summarystatistics of event attributes.

The GUI of FIG. 4C also includes a user-interface element 440 (e.g., atext box) for obtaining an aggregation interval over which eventattributes are to be aggregated into a single aggregated event. Theaggregation interval may be increased to increase the amount ofaggregation in the aggregated event stream and reduced to decrease theamount of aggregation in the aggregated event stream.

For example, column 438 may indicate that the “dest_ip,” “dest_port,”“src_ip,” “status,” and “uri_path” event attributes are specified as keyattributes and the “bytes,” “bytes_in,” “bytes__out,” and “time_taken”event attributes are specified as aggregation attributes. Similarly, anaggregation interval of 60 seconds may be obtained from user-interfaceelement 440. As a result, the aggregated event stream may includeaggregated events generated from event data over a 60-second interval.After each 60-second interval has passed, a separate aggregated eventwith a unique key may be generated for each unique combination of“dest_ip,” “dest_port,” “src_ip,” “status,” and “uri_path” keyattributes encountered during the interval. Values of “bytes,”“bytes_in,” “bytes_out,” and “time_taken” for events within the intervalthat match the unique combination of key attributes may also be summedand/or otherwise aggregated into the aggregated event. Aggregated eventsgenerated from the configuration options may then be shown in the sameGUI, as described in further detail below with respect to FIG. 4F.

Such configuration of event streams and/or aggregated event streams mayallow network data to be captured at different levels of granularityand/or for different purposes. For example, an aggregated event streammay include all possible event attributes for the event stream to enableoverall monitoring of network traffic. On the other hand, one or moreunaggregated event streams may be created to capture specific types ofnetwork data at higher granularities than the aggregated event stream.In addition, multiple event streams may be created from the same packetflow and/or event data to provide multiple “views” of the packet flowand/or event data.

FIG. 4D shows an exemplary screenshot in accordance with the disclosedembodiments. More specifically, FIG. 4D shows a screenshot of the GUI ofFIGS. 4B-4C after an “Add” link in column 420 is selected. For example,the GUI of FIG. 4D may be shown as an overlay on the screens of FIGS.4B-4C to enable the addition of filters to configuration information forthe event stream(s) and/or aggregated event stream(s) shown on thescreens.

As with the screenshots of FIGS. 4A-4C, the GUI of FIG. 4D includesinformation and/or user-interface elements organized into a table. Rowsof the table may represent filters for an event stream and/or aggregatedevent stream, and columns 442-450 of the table may facilitateidentification and/or configuration of the filters.

First, column 442 may provide a list of terms representing eventattributes to which the filters are to be applied. For example, column422 may specify an “http.status” term representing the “status” eventattribute and an “http.uri-stem” term representing the “uri_path” eventattribute.

Column 444 may be used to provide a comparison associated with eachfilter. For example, a user may select a cell under column 444 to accessa drop-down menu of possible comparisons for the corresponding filter.As shown in FIG. 4D, the second cell of column 444 is selected to reveala drop-down menu of comparisons for a string-based event attribute(e.g., “uri_path”). Within the drop-down menu, “Regular Expression” isselected, while other options for the comparison may include “False,”“True,” “Is defined,” “Is not defined,” “Not Regular Expression,”“Exactly matches,” “Does not exactly match,” “Contains,” “Does notcontain,” “Starts with,” “Does not start with,” “Ends with,” “Does notend with,” “Ordered before,” “Not ordered before,” “Ordered after,” and“Not ordered after.” As a result, a number of comparisons may be madewith string-based event attributes during filtering of network data bythe string-based event attributes.

Column 446 may allow the user to specify a value against which thecomparison in column 444 is made. Cells in column 446 may betext-editable fields and/or other user-interface elements that acceptuser input. For example, the second cell of column 446 may include avalue of “admin” that is entered by the user. Consequently, the valuesin the second cells of columns 444-446 may be used to generate a filterthat determines if the “uri_path” event attribute from network datamatches a regular expression of “admin.” If the network data matches theregular expression, the network data may be used to generate event data,which may subsequently be used to generate aggregated event data. If thenetwork data does not match the regular expression, generation of eventdata from the network data may be omitted.

Column 448 may include a set of checkboxes with a “Match All” header.The user may check a checkbox in column 448 to require each value in amulti-value event attribute to match the filter. For example, the usermay check a checkbox in column 448 for a filter that is applied to achecksum event attribute to ensure that each of multiple checksums in agiven network packet and/or event satisfies the comparison in thefilter.

Column 450 may allow the user to delete filters from the configurationinformation. For example, the user may select a user-interface (e.g., anicon) in a cell of column 450 to remove the corresponding filter fromthe configuration information.

The GUI also includes a set of user-interface elements 452-454 fordetermining the applicability of individual filters or all filters tothe network data. For example, the user may select user-interfaceelement 452 (e.g., “All”) to apply the filters so that only data thatmatches all filters in the table is used to generate events. Conversely,the user may select user-interface element 454 (e.g., “Any”) to applythe filters so that data matching any of the filters in the data is usedto generate events. In other words, user-interface element 452 may beselected to apply a logical conjunction to the filters, whileuser-interface element 454 may be selected to apply a logicaldisjunction to the filters.

FIG. 4E shows an exemplary screenshot in accordance with the disclosedembodiments. As with the screenshot of FIG. 4D, FIG. 4E shows a GUI foradding and/or managing filters for generating event data at one or moreremote capture components.

Within the GUI of FIG. 4E, the first cell of column 444 is selected. Inturn, a drop-down menu of possible comparisons is shown for thecorresponding filter. Because the filter relates to a numeric eventattribute (e.g., an HTTP status code), comparisons in column 444 may benumeric in nature. For example, the “Greater than” comparison isselected, while other possible comparisons may include “False,” “True,”“Is defined,” “Is not defined,” “Equals,” “Does not equal,” “Less than,”“Greater than or equal to,” and “Less than or equal to.” The differencesin comparisons shown in FIG. 4E and FIG. 4D may ensure that comparisonsthat are meaningful and/or relevant to the types of event attributesspecified in the filters are used with the filters.

FIG. 4F shows an exemplary screenshot in accordance with the disclosedembodiments. More specifically, FIG. 4F shows a screenshot of a GUI,such as GUI 325 of FIG. 3. The GUI of FIG. 4F may provide informationrelated to aggregated events, such as aggregated events generated usingthe GUI of FIG. 4C.

As shown in FIG. 4F, a first column 456 contains a timestamp of anaggregated event, and a second column 458 shows the aggregated event.Within column 458, the aggregated event includes a number of eventattributes. Some of the event attributes (e.g., “dest_ip,” “dest_port,”“src_ip,” “status,” “uri_path”) are key attributes that are used touniquely identify the aggregated event, and other event attributes(e.g., “dest_port,” “status,” “bytes,” “bytes_in,” “bytes_out,”“time_taken”) may be numerically summed before the event attributes areincluded in the aggregated event.

FIG. 5A shows an exemplary screenshot in accordance with the disclosedembodiments. More specifically, FIG. 5A shows a screenshot of a GUI,such as GUI 325 of FIG. 3. The GUI may be used with arisk-identification mechanism and/or a capture trigger, such asrisk-identification mechanism 307 and capture trigger 309 of FIG. 3.

The GUI of FIG. 5A may include a portion 502 that represents therisk-identification mechanism. For example, portion 502 may display adashboard of time-series event data that represents security risks. Thedashboard includes a number of potential security risks, such as “HTTPErrors,” “DNS Errors,” “Cloud Email,” “NFS Activity,” and “Threat ListActivity.” Events that match one of the listed potential security risksmay be represented as bars within a time interval represented by thehorizontal dimension of the dashboard. For example, a security risk 506may be shown as a series of bars clustered around an interval of timeunder “DNS Errors” in portion 502.

On the other hand, the dashboard may lack data for other potentialsecurity risks because the data volume associated with capturing networkdata across all protocols and/or security risks may be too large toeffectively store and/or consume. As a result, portion 502 may indicatethat no data is available (e.g., “Search returned no results”) for the“HTTP Errors,” “Cloud Email,” “NFS Activity,” and “Threat List Activity”security risks.

The GUI may also include a portion 504 that represents a capture triggerfor generating additional time-series event data based on identifiedsecurity risks from portion 502. For example, portion 504 may include acheckbox that allows a user to activate the capture trigger uponidentifying security risk 506 in portion 502. Portion 504 may alsoinclude a first drop-down menu that allows the user to specify one ormore protocols (e.g., “HTTP,” “DNS,” “All Email,” “NFS/SMB,” “AllProtocols”) of additional time-series event data to be captured with thecapture trigger. Portion 504 may additionally include a second drop-downmenu that allows the user to specify a period (e.g., “4 Hours”) overwhich the additional time-series event data is to be captured after thecapture trigger is activated.

After the capture trigger is activated, configuration information on oneor more remote capture agents used to generate the time-series eventdata may be updated to include the additional protocol(s) specified inportion 504. For example, configuration information for configuring thegeneration of additional event streams from the specified protocol(s)may be propagated to the remote capture agents, and the remote captureagents may use the configuration to create the event streams fromnetwork data and/or event data at the remote capture agents. Theconfiguration information may include default event attributes for theprotocol(s) and/or event attributes that may be of interest to thesecurity assessment of network packet flows. For example, theconfiguration information may specify the generation of event datarelated to other security risks, such as the security risks shown in thedashboard. Once the event data is generated and/or indexed, the eventdata may be shown in the dashboard to facilitate verification,monitoring, and/or analysis of the security risk. After thepre-specified period obtained from portion 504 has passed, theconfiguration information on the remote capture agents may be updated todisable the generation of the additional event streams and reduce thevolume of network data captured by the remote capture agents.

As with the user interfaces of FIGS. 4A-4E, the user may add one or morefilters that are applied during the generation of the additionaltime-series event data. For example, the user may use the userinterfaces of FIGS. 4D-4E to add a filter for network and/or event datathat exactly matches the IP address (e.g., 10.160.26.206) from which thesecurity risk was detected. As a result, the additional time-series datamay be generated only from network data containing the same source IPaddress. The user may also use the user interfaces of FIGS. 4A-4C tocustomize the collection of additional time-series event data byprotocol and/or event attributes.

FIG. 5B shows an exemplary screenshot in accordance with the disclosedembodiments. In particular, FIG. 5B shows a screenshot of a GUI, such asGUI 325 of FIG. 3. Like the GUI of FIG. 5A, the GUI of FIG. 5B includesa first portion 506 representing a risk-identification mechanism and asecond portion 508 representing a capture trigger.

Portion 506 may allow a user to create a recurring search fortime-series event data that matches a security risk. For example,portion 506 may include user-interface elements for obtaining a domain,application context, description, search terms, time range (e.g., startand end times), and/or frequency (e.g., daily, hourly, every fiveminutes, etc.) for the recurring search. The user may use theuser-interface elements of portion 506 to specify a recurring search foran excessive number of failed login attempts in captured network and/orevent data, which may represent brute force access behavior thatconstitutes a security risk.

Portion 508 may allow the user to provide the capture trigger, which isautomatically activated if the recurring search finds time-series eventdata that matches the security risk. As with portion 504 of FIG. 5A,portion 508 may allow the user to set the capture trigger, specify oneor more protocols to be captured with the capture trigger, and/or apre-specified period over which network data using the protocol(s) is tobe captured.

After the user has finished defining the recurring search and capturetrigger, the user may select a user-interface 510 (e.g., “Save”) to savethe recurring search and capture trigger. The capture trigger may thenbe activated without additional input from the user once an iteration ofthe recurring search identifies the security risk. Conversely, the usermay select a user-interface element 512 (e.g., “Cancel”) to exit thescreen of FIG. 5B without creating the recurring search and/or capturetrigger.

FIG. 6 shows a flowchart illustrating the processing of network data inaccordance with the disclosed embodiments. In one or more embodiments,one or more of the steps may be omitted, repeated, and/or performed in adifferent order. Accordingly, the specific arrangement of steps shown inFIG. 6 should not be construed as limiting the scope of the embodiments.

Initially, configuration information is obtained at a remote captureagent from a configuration server over a network (operation 602). Theremote capture agent may be located on a separate network from that ofthe configuration server. For example, the remote capture agent may beinstalled on a physical and/or virtual machine on a remote networkand/or cloud. As discussed above, the remote capture agent and otherremote capture agents may be used to capture network data from a set ofremote networks in a distributed manner.

Next, the configuration information is used to configure the generationof event data from network packets captured by the remote capture agentduring the runtime of the remote capture agent (operation 604). Forexample, the configuration information may be used to configure theremote capture agent to identify certain types of network packets,extract network data from the network packets, and/or include thenetwork data in the event data.

The remote capture agent may identify network packets in a packet flowbased on control information in the network packets (operation 608). Forexample, network packets between a source and destination may beidentified based on source and/or destination network addresses, sourceand/or destination ports, and/or transport layer protocols in theheaders of the network packets.

The remote capture agent may also assemble the packet flow from thenetwork packets (operation 608) and/or decrypt the network packets upondetecting encryption of the network packets (operation 610). Forexample, the remote capture agent may rearrange out-of-order TCP packetsinto a TCP stream. The remote capture agent may also analyze the bytesignatures of the network packets' payloads to identify encryption ofthe network packets and use an available private key to decrypt thenetwork packets.

After the packet flow is identified, assembled and/or decrypted, theremote capture agent may obtain a protocol classification for the packetflow (operation 612). For example, the remote capture agent may providenetwork packets in the packet flow to a protocol-decoding mechanism andreceive one or more protocol identifiers representing the protocols usedby the network packets from the protocol-decoding mechanism.

Next, the remote capture agent may use configuration informationassociated with the protocol classification to build an event streamfrom the packet flow (operation 614), as described in further detailbelow with respect to FIG. 7. The remote capture agent may then transmitthe event stream over a network for subsequent storage and processing ofthe event stream by one or more components on the network (operation616). For example, the remote capture agent may transmit the eventstream to one or more data storage servers, configuration servers,and/or indexers on the network.

An update to the configuration information may be received (operation616). For example, the remote capture agent may receive an update to theconfiguration information after the configuration information ismodified at a configuration server. If an update to the configurationinformation is received, the update is used to reconfigure thegeneration of time-series event data at the remote capture agent duringruntime of the remote capture agent (operation 620). For example, theremote capture agent may be use the updated configuration information togenerate one or more new event streams, discontinue the generation ofone or more existing event streams, and/or modify the generation of oneor more existing event streams.

The remote capture agent may continue to be used (operation 622) tocapture network data. If the remote capture agent is to be used, packetflows captured by the remote capture agent are identified (operation606), and network packets in the packet flows are assembled into thepacket flows and/or decrypted (operations 608=610). Protocolclassifications for the packet flows are also obtained and used, alongwith configuration information associated with the protocolclassifications, to build event streams from the packet flows(operations 612-614). The event streams are then transmitted over thenetwork (Operation 616), and any updates to the configurationinformation are used to reconfigure the operation of the remote captureagent (operations 618-620) during generation of the event streams.Capture of network data by the remote capture agent may continue untilthe remote capture agent is no longer used to generate event data fromnetwork data.

FIG. 7 shows a flowchart illustrating the process of using configurationinformation associated with a protocol classification to build an eventstream from a packet flow in accordance with the disclosed embodiments.In one or more embodiments, one or more of the steps may be omitted,repeated, and/or performed in a different order. Accordingly, thespecific arrangement of steps shown in FIG. 7 should not be construed aslimiting the scope of the embodiments.

First, one or more event attributes associated with the protocolclassification are obtained from the configuration information(operation 702). For example, the event attribute(s) may be obtainedfrom a portion of the configuration information that specifies thegeneration of an event stream from network data matching the protocolclassification.

Next, the event attribute(s) are extracted from network packets in thepacket flow (operation 704). For example, the event attribute(s) may beused to generate event data from the network packets. The configurationinformation may optionally be used to transform the extracted eventattribute(s) (operation 706). For example, the configuration informationmay be used to aggregate the event data into aggregated event data thatreduces the volume of event data generated while retaining the importantaspects of the event data.

Finally, the extracted and/or transformed event attributes are includedin the event stream (operation 708). For example, the event stream maybe include a series of events and/or aggregated events that containevent attributes that are relevant to the protocol classification of thenetwork packets represented by the events.

FIG. 8 shows a flowchart illustrating the process of facilitating theprocessing of network data in accordance with the disclosed embodiments.In one or more embodiments, one or more of the steps may be omitted,repeated, and/or performed in a different order. Accordingly, thespecific arrangement of steps shown in FIG. 8 should not be construed aslimiting the scope of the embodiments.

First, a GUI for obtaining configuration information for configuring thegeneration of time-series event data from network packets captured byone or more remote agents is provided (operation 802). The GUI mayinclude a number of user-interface elements for streamlining thecreation and/or update of the configuration information. First, the GUImay provide a set of user-interface elements for including one or moreevent attributes in the time-series event data of an event streamassociated with a protocol classification of the network packets(operation 804). For example, the GUI may include a set of checkboxesthat enable the selection of individual event attributes for inclusionin the time-series event data.

Second, the GUI may provide a set of user-interface elements formanaging the event stream (operation 806) and/or obtaining the protocolclassification for the event stream. For example, the GUI may includeone or more user-interface elements for cloning the event stream from anexisting event stream, which imparts the protocol classification of theexisting event stream on the cloned event stream. The GUI may alsoinclude user-interface elements for deleting the event stream, enablingthe event stream, and/or disabling the event stream.

Third, the GUI may provide a set of user-interface elements forfiltering the network packets (operation 808) prior to generating thetime-series event data from the network packets. Each filter mayidentify an event attribute, a comparison to be performed on the eventattribute, and/or a value to which the event attribute is to becompared. For example, the filter may match the event attribute to aBoolean value (e.g., true or false), perform a numeric comparison (e.g.,equals, greater, less than, greater than or equal to, less than or equalto), and/or verify the definition of (e.g., the existence of) the eventattribute in network data. The filter may also compare the eventattribute to a regular expression, perform an exact match of the eventattribute to the value, perform a partial match of the event attributeto the value, and/or determine the event attribute's position in anordering.

Fourth, the GUI may provide a set of user-interface elements foraggregating the event attribute(s) into aggregated event data that isincluded in the event stream (operation 810). For example, the GUI mayprovide user-interface elements for identifying event attributes as keyattributes used to generate a key representing the aggregated event dataand/or aggregation attributes to be aggregated prior to inclusion in theaggregated event data. The GUI may also include one or moreuser-interface elements for obtaining an aggregation interval over whichthe one or more event attributes are aggregated into the aggregatedevent data.

Finally, the event attribute(s), protocol classification, filteringinformation, and/or aggregation information obtained from the GUI areincluded in the configuration information (operation 812). Theconfiguration information may then be used to configure theprotocol-based capture, filtering, and/or aggregation of network data atthe remote capture agent(s).

FIG. 9 shows a flowchart illustrating the process of facilitating theprocessing of network data in accordance with the disclosed embodiments.In one or more embodiments, one or more of the steps may be omitted,repeated, and/or performed in a different order. Accordingly, thespecific arrangement of steps shown in FIG. 9 should not be construed aslimiting the scope of the embodiments.

Initially, a risk-identification mechanism for identifying a securityrisk from time-series event data generated from network packets capturedby one or more remote capture agents distributed across a network isprovided (operation 902). The risk-identification mechanism may includea GUI that displays an event of interest related to the security risk.For example, the GUI may show potential security risks in a dashboardand/or other visualization of the time-series event data. Alternatively,the risk-identification mechanism may include a search and/or recurringsearch for a subset of the time-series event data matching the securityrisk. For example, the risk-identification mechanism may include asearch mechanism that allows a user to search for threats, attacks,errors, and/or other notable events in the time-series event data.

Next, a capture trigger for generation additional time-series data fromthe network packets on the remote capture agent(s) based on the securityrisk is provided (operation 904). The capture trigger may be receivedthrough one or more user-interface elements of a GUI, such as the sameGUI used to provide the risk-identification mechanism. For example, thecapture trigger may be activated in a portion of the GUI that is above,below, and/or next to a dashboard that displays security risks to theuser. Alternatively, the capture trigger may be linked to a recurringsearch for time-series event data that matches a security risk. As aresult, the capture trigger may automatically be activated oncetime-series event data matching the security risk is found.

After the capture trigger is activated, the capture trigger is used toconfigure the generation of the additional time-series event data fromthe network packets (operation 906). For example, activation of thecapture trigger may result in the updating of configuration informationfor the remote capture agent(s), which causes the remote captureagent(s) to generate additional event streams containing eventattributes associated with protocols that facilitate analysis of thesecurity risk.

Finally, generation of the additional time-series event data is disabledafter a pre-specified period has passed (operation 908). For example,generation of the additional time-series event data may be set to expirea number of hours or days after the capture trigger is activated. Theexpiry may be set by the user and/or based on a default expiration forsecurity-based capture of additional network data from network packets.

FIG. 10 shows a computer system 1000 in accordance with the disclosedembodiments. Computer system 1000 includes a processor 1002, memory1004, storage 1006, and/or other components found in electroniccomputing devices. Processor 1002 may support parallel processing and/ormulti-threaded operation with other processors in computer system 1000.Computer system 1000 may also include input/output (I/O) devices such asa keyboard 1008, a mouse 1010, and a display 1012.

Computer system 1000 may include functionality to execute variouscomponents of the disclosed embodiments. In particular, computer system1000 may include an operating system (not shown) that coordinates theuse of hardware and software resources on computer system 1000, as wellas one or more applications that perform specialized tasks for the user.To perform tasks for the user, applications may obtain the use ofhardware resources on computer system 1000 from the operating system, aswell as interact with the user through a hardware and/or softwareframework provided by the operating system.

In one or more embodiments, computer system 1000 provides a system forfacilitating the processing of network data. The system may include aremote capture agent. The remote capture agent may obtain a firstprotocol classification for a first packet flow captured at the remotecapture agent and use configuration information associated with thefirst protocol classification to build a first event stream from thefirst packet flow at the remote capture agent. The remote capture agentmay also transmit the first event stream over a network for subsequentstorage and processing of the first event stream by one or morecomponents on the network. The remote capture agent may further obtain asecond protocol classification for a second packet flow captured at theremote capture agent, use configuration information associated with thesecond protocol classification to build a second event stream from thesecond packet flow at the remote capture agent, and transmit the secondevent stream over the network.

The system may also provide a configuration server that provides theconfiguration information to the remote capture agent. The configurationserver may provide a GUI for obtaining configuration for configuring thegeneration of time-series event data from network packets captured bythe remote capture agent. The GUI may include user-interface elementsfor including one or more event attributes in the time-series event dataof an event stream associated with a protocol classification of thenetwork packets. The GUI may also include user-interface elements formanaging the event stream, filtering the network packets, and/oraggregating the event attributes into aggregated event data.

The system may additionally provide a risk-identification mechanism foridentifying a security risk from the time-series event data generated bythe remote capture agent. Finally, the system may provide a capturetrigger for generating additional time-series event data from thenetwork packets on the remote capture agent based on the security risk.The additional time-series data may include one or more event attributesfor facilitating analysis of the security risk, such as an eventattribute associated with a protocol that facilitates analysis of thesecurity risk.

The foregoing descriptions of various embodiments have been presentedonly for purposes of illustration and description. They are not intendedto be exhaustive or to limit the present invention to the formsdisclosed. Accordingly, many modifications and variations will beapparent to practitioners skilled in the art. Additionally, the abovedisclosure is not intended to limit the present invention.

What is claimed is:
 1. A computer-implemented method comprising:receiving, by a remote capture agent, configuration information forgenerating time-stamped events based on network traffic monitored by theremote capture agent; identifying a packet flow from the networktraffic; determining that network packets of the packet flow areencrypted network packets; obtaining decrypted network packets bydecrypting the encrypted network packets; generating, based on theconfiguration information, an event stream comprising time-series eventdata including data included in the decrypted network packets; andtransmitting the event stream to another component on the network. 2.The computer-implemented method of claim 1, wherein determining that thenetwork packets of the packet flow are encrypted network packetsincludes analyzing byte signatures associated with the network packets.3. The computer-implemented method of claim 1, further comprisingobtaining an encryption key from a Secure Sockets Layer (SSL) server,wherein the encrypted network packets are decrypted using the encryptionkey.
 4. The computer-implemented method of claim 1, wherein the remotecapture agent generates the event stream in part by: extracting an eventattribute from a decrypted network packet of the decrypted networkpackets, and generating a timestamped event including the eventattribute.
 5. The computer-implemented method of claim 1, wherein theremote capture agent generates the event stream in part by: for aplurality of the decrypted network packets associated with a definedinterval of time, extracting an event attribute from the plurality ofthe decrypted network packets to obtain a plurality of extracted eventattribute values; generating aggregated event data based on theplurality of extracted event attribute values; and generating anaggregated event including the aggregated event data.
 6. Thecomputer-implemented method of claim 1, further comprising sending, bythe remote capture agent, the event stream to another component on thenetwork for subsequent processing.
 7. The computer-implemented method ofclaim 1, further comprising extracting an event attribute from adecrypted network packet of the decrypted network packets, wherein theevent attribute is one of: a source address, a destination address, anetwork address, or a port.
 8. The computer-implemented method of claim1, further comprising: identifying a network protocol associated withthe packet flow; identifying, based on the configuration information, anevent attribute associated with the network protocol; and extracting theevent attribute from a decrypted network packet of the decrypted networkpackets.
 9. The computer-implemented method of claim 1, wherein theconfiguration information specifies a transformation to be applied tothe time-series event data to be included in the event stream, andwherein the method further comprises transforming the time-series eventdata according to the transformation.
 10. The computer-implementedmethod of claim 1, wherein the remote capture agent is installed in acloud computing environment.
 11. A non-transitory computer-readablestorage medium storing instructions which, when executed by a processor,cause the processor to perform operations comprising: receiving, by aremote capture agent, configuration information for generatingtime-stamped events based on network traffic monitored by the remotecapture agent; identifying a packet flow from the network traffic;determining that network packets of the packet flow are encryptednetwork packets; obtaining decrypted network packets by decrypting theencrypted network packets; generating, based on the configurationinformation, an event stream comprising time-series event data includingdata included in the decrypted network packets; and transmitting theevent stream to another component on the network.
 12. The non-transitorycomputer-readable storage medium of claim 11, wherein determining thatthe network packets of the packet flow are encrypted network packetsincludes analyzing byte signatures associated with the network packets.13. The non-transitory computer-readable storage medium of claim 11,further comprising obtaining an encryption key from a Secure SocketsLayer (SSL) server, wherein the encrypted network packets are decryptedusing the encryption key.
 14. The non-transitory computer-readablestorage medium of claim 11, wherein the remote capture agent generatesthe event stream in part by: extracting an event attribute from adecrypted network packet of the decrypted network packets, andgenerating a timestamped event including the event attribute.
 15. Thenon-transitory computer-readable storage medium of claim 11, wherein theremote capture agent generates the event stream in part by: for aplurality of the decrypted network packets associated with a definedinterval of time, extracting an event attribute from the plurality ofthe decrypted network packets to obtain a plurality of extracted eventattribute values; generating aggregated event data based on theplurality of extracted event attribute values; and generating anaggregated event including the aggregated event data.
 16. A systemcomprising: a first one or more electronic devices to implement a remotecapture agent, wherein remote capture agent includes first instructionsthat upon execution cause the remote capture agent to: receiveconfiguration information for generating time-stamped events based onnetwork traffic monitored by the remote capture agent, identify a packetflow from the network traffic, determine that network packets of thepacket flow are encrypted network packets, obtain decrypted networkpackets by decrypting the encrypted network packets, generate, based onthe configuration information, an event stream comprising time-seriesevent data including data included in the decrypted network packets, andtransmit the event stream to another component on the network; and asecond one or more electronic devices to implement a data storageserver, wherein the data storage server includes second instructionsthat upon execution cause the data storage server to: receive the eventstream from the remote capture agent, and store the event stream. 17.The system of claim 16, wherein determining that the network packets ofthe packet flow are encrypted network packets includes analyzing bytesignatures associated with the network packets.
 18. The system of claim16, wherein the remote capture agent further includes first instructionsthat upon execution cause the remote capture agent to obtain anencryption key from a Secure Sockets Layer (SSL) server, wherein theencrypted network packets are decrypted using the encryption key. 19.The system of claim 16, wherein the remote capture agent generates theevent stream in part by: extracting an event attribute from a decryptednetwork packet of the decrypted network packets, and generating atimestamped event including the event attribute.
 20. The system of claim16, wherein the remote capture agent generates the event stream in partby: for a plurality of the decrypted network packets associated with adefined interval of time, extracting an event attribute from theplurality of the decrypted network packets to obtain a plurality ofextracted event attribute values; generating aggregated event data basedon the plurality of extracted event attribute values; and generating anaggregated event including the aggregated event data.